Operating modes that designate an interface modality for interacting with an automated assistant

ABSTRACT

Implementations described herein relate to transitioning a computing device between operating modes according to whether the computing device is suitably oriented for received non-audio related gestures. For instance, the user can attach a portable computing device to a docking station of a vehicle and, while in transit, wave their hand near the portable computing device in order to invoke the automated assistant. Such action by the user can be detected by a proximity sensor and/or any other device capable of determining a context of the portable computing device and/or an interest of the user in invoking the automated assistant. In some implementations location, orientation, and/or motion of the portable computing device can be detected and used in combination with an output of the proximity sensor to determine whether to invoke the automated assistant in response to an input gesture from the user.

BACKGROUND

Humans may engage in human-to-computer dialogs with interactive softwareapplications referred to herein as “automated assistants” (also referredto as “digital agents,” “chatbots,” “interactive personal assistants,”“intelligent personal assistants,” “assistant applications,”“conversational agents,” etc.). For example, humans (which when theyinteract with automated assistants may be referred to as “users”) mayprovide commands and/or requests to an automated assistant using spokennatural language input (i.e. utterances), which may in some cases beconverted into text and then processed, and/or by providing textual(e.g., typed) natural language input. An automated assistant responds toa request by providing responsive user interface output, which caninclude audible and/or visual user interface output.

In many cases, before the automated assistant can interpret and respondto a user's request, it must first be “invoked,” for example, usingpredefined oral invocation phrases that are often referred to as “hotwords” or “trigger phrase.” Thus, many automated assistants operate inwhat will be referred to herein as a “default listening state” in whichthey are always “listening” to audio data sampled by a microphone for alimited (or finite, or “default”) set of hot words. Any utterancescaptured in the audio data other than the default set of hot words areignored. Once the automated assistant is invoked with one or more of thedefault set of hot words, it may operate in what will be referred toherein as a “speech recognition state” wherein for at least some timeinterval after invocation, the automated assistant performsspeech-to-text (“STT”) processing of audio data sampled by a microphoneto generate textual input, which in turn is semantically processed todetermine a user's intent (and to fulfill that intent).

However, in some contexts, a user may not be able to provide a coherent“hot word,” such as when the user is driving in a car where multiplepeople are chatting, or when the user is at home listening to music orwatching a movie. Should the automated assistant attempt to process a“hot word” in such contexts, the automated assistant may inadvertentlycapture audio being concurrently provided from a variety of differentsources. As a result, the automated assistant may fail and/or provideerroneous output when a “hot word” is not discernible from other soundsthat have been captured by audio data received by the automatedassistant. This can lead to excess usage of computational and/or networkresources, for instance, as a result of a user needing to provide asupplemental invocation phrase, that is more discernible from otherbackground noise. Such supplemental invocation phrases must additionallybe processed by a corresponding client device and/or remote automatedassistant component(s), thereby causing additional usage of variouscomputational/network resources.

SUMMARY

Implementations set forth herein relate to invoking an automatedassistant from a computing device using a speechless and/or inaudiblephysical gesture, depending upon whether the computing device isoperating in a particular environment and/or under particularconditions. In certain conditions, a user may not be able to clearlyprovide a trigger phrase because of background noise created by otherpeople that are talking around the user, other devices that areprojecting audio toward the user, and/or any other audible sounds thatcan interrupt or interfere with speech from the user. As a result, theautomated assistant may end up processing audio that was not intendedfor the automated assistant, causing the automated assistant to providean inadequate response, thereby wasting computational and/or networkresources. For example, computational and/or network resources can bewasted when audio that has been captured by an assistant device istransmitted to a remote server for processing, and the processing of theaudio data does not result in any content that is discernible by theautomated assistant. Furthermore, should the user have to repeat atrigger phrase, in furtherance of ensuring that the automated assistantreceives some amount of discernible audio input, such repetition of thesame input to the automated assistant can waste power and computationalresources, as the device that is providing access to the automatedassistant must continually monitor and process audible input from theuser. In order to resolve such technical deficiencies, theimplementations set forth herein allow a user to invoke the automatedassistant in response to a device detecting a non-audible feature of aphysical gesture performed by the user, at least when the device isoperating under particular conditions. In this way, should the user bein an environment where providing a clear spoken utterance would bedifficult, the user can simply perform a particular physical gesture inorder to initially invoke the automated assistant, despite the noisyenvironment.

In some implementations, the automated assistant can be accessible to aportable computing device, such as a cellular phone, which can operateaccording to multiple different operating modes for invoking theautomated assistant. For example, a first operating mode can allow theuser to invoke the automated assistant by providing a spoken utteranceto an automated assistant interface of the portable computing device.The portable computing device can transition into a second operatingmode, in which the automated assistant can be invoked upon detection ofa non-audio feature of a physical gesture performed by the user. Thephysical feature can be, for example, a hand wave over a portion of theportable computing device. The hand wave can be detected by a proximitysensor, which can be considered as a sensor of the one or more sensorsthat comprise the automated assistant interface.

The portable computing device can transition between the first operatingmode and the second operating mode based on various data that isaccessible to the portable computing device. In some implementations,the various data can include contextual data that characterizes acontext in which the portable computing device is currently, or haspreviously been, operating. Alternatively, or additionally, the variousdata that is used to determine whether to transition the portablecomputing device from the first operating mode to the second operatingmode can be provided by a device that is separate from the portablecomputing device. For example, a device that is separate from theportable computing device can be a docking apparatus, which can beconfigured to provide at least some amount of mechanical support to aportion of the portable computing device. Furthermore, in someimplementations, the docking apparatus can include a transmitter, suchas a near field communication (NFC) transmitter, which can transmit datato the portable computing device and/or receive data from the portablecomputing device. Such data received from the docking apparatus canindicate whether the portable computing device has been docked with thedocking apparatus, as well as other data that can be associated with thedocking of the portable computing device. In this way, data provided bythe docking apparatus can be processed by the portable computing device,or a device that is in communication with the portable computing device,in order to determine whether to transition the portable computingdevice from the first operating mode to the second operating mode.

In some implementations, the docking apparatus can be connected to avehicle associated with the user. The user can conveniently dock theportable computing device with the docking apparatus in order that theuser can receive content from the portable computing device whiledirecting more of their attention to driving their vehicle, or otherwiseperforming some activity while the vehicle drives itself. When the useris operating a vehicle in which multiple people are having aconversation, it may be difficult for the user to invoke the automatedassistant using a spoken utterance, at least as a result of noiseinterfering with the spoken utterance. Therefore, according toimplementations discussed herein, the portable computing device can becaused to transition from a first operating mode to a second operatingmode at least based on the portable computing device determining that ithas been docked with the docking apparatus. In some implementations,other information can be used in combination with data received from thedocking apparatus in order to determine whether to transition theportable computing device from the first operating mode to the secondoperating mode. The other information can include location data and/orother sensor data, which can indicate a location of a portable computingdevice, whether the portable computing device is located near the user,whether the portable computing device has received any physical contactfrom the user within a threshold period of time, whether a particularapplication is active at the portable computing device or another devicethat is associated with the portable computing device, and/or any otherrelevant information suitable for determining whether to transition aportable computing device from the first operating mode to the secondoperating mode.

In some scenarios, it may be desirable for the user to stop theautomated assistant from providing further audible output, such as whenthe automated assistant is projecting audio over a conversation or phonecall. Therefore, despite having invoked the automated assistant throughperformance of an inaudible physical gesture, the user can stop theautomated assistant from providing further audible output by performinganother inaudible physical gesture. For instance, the user can be ridingin a vehicle where the portable computing device is docked and thereforeoperating according to the second operating mode. If the automatedassistant is providing audible output when another user in the carreceives an incoming phone call, the user can pause or stop theautomated assistant from providing further audible output by performingan inaudible physical gesture. In this way, not only is the usercontrolling the automated assistant to prevent it from interrupting thephone call, but the user is also avoiding individually interrupting thephone call with a spoken utterance. In other words, the user can simplyperform an inaudible physical gesture, such as waving their hand infront of the portable computing device, in order that no additionalaudio is inserted into the environment where the other person isreceiving or participating in a phone call.

The above description is provided as an overview of some implementationsof the present disclosure. Further description of those implementations,and other implementations, are described in more detail below.

In some implementations, a method implemented by one or more processorsis set forth as including operations such as causing a portablecomputing device to operate in a first operating mode, wherein, when theportable computing device is operating in the first operating mode, theportable computing device is configured to: invoke an automatedassistant in response to detection of a spoken trigger phrase, andrestrict usage of a proximity sensor, which is in communication with theportable computing device, for invoking the automated assistant. Themethod can further include can operation of processing contextual datathat characterizes a current context of the portable computing device,wherein the contextual data is generated based on sensor output from oneor more sensors of the portable computing device. The method can alsoinclude causing the portable computing device to operate in a secondoperating mode in response to the processing of the contextual datasatisfying one or more criteria, wherein, when the portable computingdevice is operating in the second operating mode, the portable computingdevice is configured to invoke the automated assistant in response tothe proximity sensor indicating that a physical gesture was performed bya user. The method can further include, when the portable computingdevice is operating in the second operating mode and the proximitysensor of the portable computing device indicates that the physicalgesture was performed by the user: causing the automated assistant to beinvoked in response to the proximity sensor indicating that the physicalgesture was performed by the user.

In some implementations, the one or more sensors include a transmitterthat is connected to docking apparatus and wherein processing thecontextual data further includes: determining that the portablecomputing device is within a minimal distance for receiving broadcasteddata from the transmitter. In some implementations, the portablecomputing device includes a touch sensor and wherein processing thecontextual data includes: determining that the touch sensor has notreceived direct physical contact from the user for a threshold period oftime. In some implementations, processing the contextual data includesdetermining, based on the sensor output, that the portable computingdevice has been stationary at a location that is physically separatedfrom the user for a threshold period of time. In some implementations,processing the contextual data includes determining, based on processingthe contextual data, that an environment of the portable computingdevice is receiving audio being projected by another person. In someimplementations, causing the portable computing device to operate in thesecond operating mode is further in response to determining that theenvironment of the portable computing device is receiving audio that isbeing projected by the other person. In some implementations, processingthe contextual data includes determining, based on processing thecontextual data, that an environment of the portable computing device isreceiving audio that is being projected by another computing device,wherein causing the portable computing device to operate in the secondoperating is further in response to determining that the environment ofthe portable computing device is receiving audio that is being projectedby the other computing device, and wherein the automated assistant isaccessible via the other computing device. In some implementations, themethod can include causing, in response to the proximity sensorindicating that the physical gesture was performed by the user, a touchscreen display of the portable computing device to provide aninteractive graphical element that includes natural language text,wherein the interactive graphical element is configured to modify anoperation of the other computing device in response to the userproviding a different gesture for selecting the interactive graphicalelement. In some implementations, the other computing device isperforming the operation when the portable computing device is operatingin the second operating mode.

In other implementations, a method implemented by one or more processorsis set forth as including operations such as receiving, by a portablecomputing device that is operating according to a first operating mode,a near-field communication (NFC) transmission from a docking apparatus,wherein the NFC transmission indicates a presence of the dockingapparatus to the portable computing device when the portable computingdevice is within a minimal distance for detecting a broadcast from thedocking apparatus, and wherein, when the portable computing device isoperating in the first operating mode, the portable computing device isconfigured to invoke an automated assistant in response to a spokenutterance from a user. The method can further include operations such asdetermining, in response to receiving the NFC transmission, that theportable computing device is docked at the docking apparatus, which isconfigured to provide mechanical support to at least a portion of theportable computing device. The method can also include causing, inresponse to determining that the portable computing device is docked atthe docking apparatus, the portable computing device to operateaccording to a second operating mode, wherein, when the portablecomputing device is operating according to the second operating mode,the portable computing device is configured to invoke the automatedassistant in response to detecting a non-audio feature of a physicalgesture performed by the user. The method can further include, when theportable computing device determines that the non-audio feature of thephysical gesture was performed by the user: causing the automatedassistant to perform certain processing of audio data, captured by amicrophone of the portable computing device, wherein the certainprocessing is not performed when the automated assistant is not invoked.

In some implementations, the method can include, when the portablecomputing device determines that the non-audio feature of the physicalgesture was performed by the user: determining that the user hasprovided a spoken natural language input to an automated assistantinterface of the portable computing device subsequent to the userperforming the physical gesture. In some implementations, the physicalgesture is detected by a proximity sensor that is integral to theportable computing device and the spoken natural language input isdetected by the microphone. In some implementations, the method caninclude determining that contextual data, which characterizes a contextof the portable computing device, satisfies a criterion fortransitioning the portable computing device from the first operatingmode to the second operating mode, wherein causing the portablecomputing device to operate according to the second operating mode isfurther in response to determining that the contextual data satisfiesthe criterion. In some implementations, the context is a mode oftransportation in which the docking apparatus is being carried, and thecriterion identifies an automobile as at least one mode oftransportation in which the portable computing device, when docked, cantransition from the first operating mode to the second operating mode.

In some implementations, the method can include causing a portablecomputing device to operate according to a first operating mode in whichthe portable computing device is configured to invoke an automatedassistant in response to a user providing a spoken utterance. The methodcan also include receiving, while the portable computing device isoperating in the first operating mode, data indicating that the portablecomputing device is physically situated such that a proximity sensor,which is integral to the portable computing device, is capable ofdetecting a physical gesture performed by the user. The method canfurther include causing, in response to receiving the data, the portablecomputing device to operate according to a second operating mode inwhich the automated assistant is configured to be invoked in response tothe proximity sensor detecting a non-audio feature of the physicalgesture performed by the user. The method can also include, when theproximity sensor detects the non-audio feature of the physical gestureperformed by the user: causing the automated assistant to provide anatural language output via an automated assistant interface of theportable computing device

In some implementations, the method can include monitoring by theportable computing device a sensor output of the proximity sensor morefrequently when operating in the second operating mode than the firstoperating mode. In some implementations, the natural language output isat least a portion of audible dialog and the automated assistantinterface is a speaker that is connected to the portable computingdevice. In some implementations, the natural language output is textthat is provided at a selectable element, which is displayed at a touchscreen display panel of the portable computing device. In someimplementations, the data indicating that the portable computing deviceis physically situated such that the proximity sensor is capable ofdetecting the physical gesture performed by the user includes locationdata that characterizes a location of the portable computing devicerelative to the user or idle time data that characterizes an amount oftime that the user has ceased providing an input to the portablecomputing device. In some implementations, the data indicating that theportable computing device is physically situated such that the proximitysensor is capable of detecting the physical gesture performed by theuser includes trajectory data that characterizes a velocity or adirection in which the portable computing device is moving. In someimplementations, the method can include monitoring a sensor output ofthe proximity sensor to determine whether the non-audio feature of thephysical gesture has been performed by the user, wherein the non-audiofeature of the physical gesture includes positioning an appendage of theuser proximate enough to the portable computing device to cause a changein the sensor output of the proximity sensor.

In some implementations, an automated assistant can be invoked inresponse to detecting a user gesture and a user-directed gaze, and/ordetecting an occurrence of one or more other condition(s), when acorresponding computing device is operating in a particular operatingmode. The occurrence of the one or more other conditions can include,for example: detecting, based on audio data, voice activity (e.g., anyvoice activity, voice activity of the user providing the gesture anddirected gaze, voice activity of an authorized user, voice activity thatincludes a spoken invocation phrase) in temporal proximity to thedetected gesture and directed gaze; detecting, based on vision data,mouth movement of the user that co-occurs with, or is in temporalproximity to, the detected gesture and directed gaze; detecting, basedon audio data and/or vision data, that the user is an authorized user;and/or detecting other condition(s). For example, buffered audio datacan be transmitted by the client device to one or more remote automatedassistant components in response to detecting the gesture and thedirected gaze, and in response to detecting voice activity in at leastpart of the buffered audio data (e.g., using a voice activity detector(VAD) module).

Other implementations may include a non-transitory computer readablestorage medium storing instructions executable by one or more processors(e.g., central processing unit(s) (CPU(s)), graphics processing unit(s)(GPU(s)), and/or tensor processing unit(s) (TPU(s)) to perform a methodsuch as one or more of the methods described above and/or elsewhereherein. Yet other implementations may include a system of one or morecomputers and/or one or more robots that include one or more processorsoperable to execute stored instructions to perform a method such as oneor more of the methods described above and/or elsewhere herein.

It should be appreciated that all combinations of the foregoing conceptsand additional concepts described in greater detail herein arecontemplated as being part of the subject matter disclosed herein. Forexample, all combinations of claimed subject matter appearing at the endof this disclosure are contemplated as being part of the subject matterdisclosed herein.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A and FIG. 1B illustrate a computing device being transitionedinto an operating mode via which an automated assistant can be invokedusing a physical gesture that has a non-audio feature.

FIG. 2A and FIG. 2B illustrate perspective views of a user invoking anautomated assistant using a physical gesture that includes an inaudiblefeature and/or characteristic.

FIG. 3 illustrates a system for providing a client device with anability to receive non-audio related gestures from a user forcontrolling certain aspects of the client device and/or any other devicethat can be associated with the user.

FIG. 4 illustrates a method for transitioning a computing device betweenoperating modes based on whether the computing device is docked at adocking apparatus.

FIG. 5 illustrates in method for causing an automated assistant toperform a particular action when a computing device is operating in amode via which the automated assistant is capable of being responsive tonon-audio feature(s) of a physical gesture.

FIG. 6 is a block diagram of an example computer system.

DETAILED DESCRIPTION

FIG. 1A and FIG. 1B illustrate a computing device 110 being transitionedinto an operating mode via which an automated assistant can be invokedusing a physical gesture that has a non-audio feature. Specifically,FIG. 1A illustrates a perspective view 100 of a first user 102 and asecond user 104 sitting in a vehicle 106. The first user 102 can restthe computing device 110 at a location within the vehicle 106, forinstance, on a surface of the vehicle 106, and/or secure the computingdevice 110 to a docking apparatus that is attached or otherwise locatedin the vehicle 106. The computing device 110 can include or access anautomated assistant 112, which can be responsive to natural languageinputs provided by the first user 102 and/or the second user 104. Insome implementations, while the computing device 110 is located withinthe vehicle, the computing device 110 can transition into differentoperating modes, which can allow the automated assistant to be invokedin response to one or more particular gestures performed by the user102.

The computing device 110 can operate according to multiple differentoperating modes, and each operating mode can provide at least onedifferent way to invoke the automated assistant. In some implementation,an operating mode of the computing device 110 can be selected by a modeselection engine 114, which can process sensor output from one or moresensors 116 that are integral to the computing device 110.Alternatively, or additionally, the mode selection engine 114 can causethe computing device 110 to transition into a particular operating modebased on data received from one or more devices than are external to thecomputing device 110. For example, the vehicle 106 can include acomputing device and/or one or more sensors, which can provide data tothe computing device 110. Alternatively, or additionally, the user 102can have one or more wearable devices that can provide data to thecomputing device 110 for use by the mode selection engine 114 whendetermining a particular mode to transition the computing device 110.

When the user 102 is traveling in the vehicle 106 and has placed thecomputing device 110 away from their body, the user 102 may wish toinvoke the automated assistant 112 for performing a particular action,such as reading a schedule, sending a message, placing a phone call,searching the internet, and/or any other action that can be performeddirectly or indirectly by an automated assistant. However, because thefirst user 102 is riding in the vehicle 106 with the second user 104,the user 102 may not want to disturb the second user 104, who may beusing another computing device 108 to perform a particular action suchas placing a phone call or watching a movie. Furthermore, if the firstuser 102 was to provide a spoken utterance to invoke the automatedassistant 112, audio data, generated in response to the computing device110 receiving the spoken utterance, can exhibit interference caused byother sounds within or outside the vehicle 106. As a result,computational resources and/or network resources can be wasted onprocessing audio that may not be discernible by a device that isprocessing the audio. In order to eliminate the waste of suchcomputational resources and network resources, the mode selection engine114 can cause the computing device 110 to operate according to aparticular operating mode when one or more sensors 116 indicate that thecomputing device 110 is physically separated from the user 102, locatedremotely from the user 102, has not received physical contact from theuser 102 with a threshold period of time, and/or is otherwise operatingwithin a context in which a user may not be able to adequately providethe spoken utterance.

In some implementations, a docking apparatus that is included in thevehicle 106 can provide an NFC signal to the computing device 110 inorder to indicate that the computing device 110 has been docked to thedocking apparatus. In response to the computing device 110 receiving theNFC signal from the docking apparatus, the mode selection engine 114 cantransition the computing device 110 from a first operating mode to asecond operating mode. When operating the first operating mode, thecomputing device 110 can cause the automated assistant 112 to be invokedin response to a spoken utterance from the user 102. When operating inthe second operating mode, the computing device 110 can cause theautomated assistant 112 to be invoked in response to the user 102providing a physical gesture, such as a hand wave 118 over a portion ofthe computing device 110.

In some implementations, the mode selection engine 114 can transitionthe computing device 110 from the first operating mode to a secondoperating mode according to other data that is accessible to thecomputing device 110. For example, the mode selection engine 114 cancause the computing device 110 to transition into from the firstoperating mode to the second operating mode when the computing device110 connects to a local network connection that is provided by a devicethat is connected to, or integral to, the vehicle 106. Additionally, oralternatively, the mode selection engine 114 can cause the computer 110to transition from the first operating mode the second operating modewhen one or more sensors 116 indicate or provide a sensor output thatindicates the computing device 110 is traveling in a vehicle 106, iswithin an environment that includes multiple people, is within anenvironment where audio from multiple different audio sources is beingprojected, and/or is otherwise located away from the user 102 at adistance at which the user may not be able to adequately provide aspoken utterance for invoking the automated assistant 112. For example,the mode selection engine 114 can cause the computing device 110 totransition from a first operating mode to a second operating mode inresponse to determining that at least one other person besides the user102 is causing audio to be projected at a loudness level that satisfiesa criterion for transitioning to the second operating mode.Alternatively, or additionally, the mode selection engine 114 can causethe computing device 110 to transition from the first operating mode tothe second operating mode in response to determining that a touchsensor, which is integral to the computing device 110, has not receivedphysical contact from the user 102 for at least a threshold period oftime, thereby satisfying a criterion for transitioning to the secondoperating mode.

FIG. 2A and FIG. 2B illustrate a perspective view 200 and a perspectiveview 202, respectively, of a user 208 invoking an automated assistantusing a physical gesture 220. Specifically, FIG. 1A illustrates aperspective view 200 of a user 208 who has positioned a portablecomputing device 204 at a docking apparatus 218 that is located withinan environment 210, such as a kitchen. The portable computing device 204can be, for example, a tablet computing device, a cellular phone, alaptop computing device, and/or any other device that can be connectedto a docking apparatus. The portable computing device 204 can include alocal automated assistant application and/or access a server automatedassistant 226 via a network connection, such as the internet. The serverautomated assistant 226 can be provided at a server device 224 and/orany other remote computing device 222. When the user 208 provides aspoken utterance or other natural language input to the automatedassistant via the portable computing device 204, data that characterizesthe natural language input can be transmitted to the server device 224for processing, and any resulting data can be transmitted back to theportable computing device 204.

In response to the portable computing device 204 connecting to thedocking apparatus 218, the portable computing device 204 can transitionfrom a first operating mode to a second operating mode, as discussedherein. In this way, the portable computing device 204 can receiveadditional gestures 220 for invoking or otherwise controlling theautomated assistant. In some implementations, the docking apparatus 218can communicate with the portable computing device 204 to indicate tothe portable computing device 204 that the user has docked the portablecomputing device 204 at the docking apparatus 218. In someimplementations, data transmitted by the docking apparatus 218 can beused in combination with data, which characterizes a context in whichthe portable computing device 204 is docked, to determine whether totransition a portable computing device 204 from the first operating modeto the second operating mode. For example, contextual data cancharacterize an amount of audio being projected from one or more otherdevices that are different from the portable computing device 204.Specifically, if a client device 206, such as a standalone speakerdevice, is playing music 212, the additional contextual data cancharacterize a level at which the client device 206 is playing music. Ifa loudness (e.g., measured or proportional to Decibels) or an amount ofnoise projected by the client device 206 reaches or exceeds a thresholdlevel, or otherwise satisfies a criterion, the portable computing device204 can transition from a first operating mode to a second operatingmode.

In some implementations, a physical gesture 220 being performed by theuser 208 can detected by the portable computing device 204, when theportable computing device 204 is docked at the docking apparatus 218.For instance, the user 208 may not be able to adequately provide aspoken utterance for invoking the automated assistant via an automatedassistant interface of the portable computing device 204. Therefore,because the user 208 may have multiple devices through which to accessthe automated assistant, the user 208 can provide a physical gesturethat includes a non-audio feature for invoking an automated assistant214 at the client device 206. Specifically, when the portable computingdevice 204 is docked at the docking apparatus 218, or otherwiseoperating in the second operating mode, the user 208 can provide aphysical gesture 220, such as a hand wave, in front of a camera or overa proximity sensor of the portable computing device 204, in order toindicate to the portable computing device 204 that the user 208 wouldlike to invoke an automated assistant. In response to the portablecomputing device 204 acknowledging or detecting the physical gesture220, the portable computing device 204 can communicate with the serverdevice 224 or the client device 206 to indicate that the user 208 isattempting to invoke the automated assistant. In response, the serverdevice 224 can communicate with the client device 206 in order to invokethe client automated assistant 214. Alternatively, or additionally, inresponse to the portable computing device 204 communicating with theclient device 206, the client device 206 can invoke the client automatedassistant 214. In this way, although the client device 206 includes anassistant interface 216 with which the user 208 can provide a spokenutterance to invoke the client automated assistant 214, the user 208 canalso rely on a non-audio feature of a physical gesture 220 in order toinvoke the client automated assistant 214 via the portable computingdevice 204.

In some implementations, the portable computing device 204 cantransition from a first operating mode to a second operating mode basedon an operating status of the client device 206. For example, when theclient device 206 begins playing music 212, data that characterizes anoperation of the client device 206 can be transmitted to the serverdevice 224 and/or portable computing device 204. In response to theserver device 224 and/or the portable computing device 204 receiving thedata, the portable computing device 204 can transition from the firstoperating mode to the second operating mode. In some implementations, adetected operating status of the client device 206 can cause theportable computing device 204 to transition from the first operatingmode to the second operating mode. An operating status can include astatus indicating that the client device 206 is participating in a phonecall, playing music or other audio, conducting a query at the request ofthe user 208, receiving a natural language input from the user 208,providing a natural language output for the user 208, and/or otherwiseperforming an action that may interfere with an ability of the portablecomputing device 204 to acknowledge a spoken utterance from the user208.

In some implementations, when the portable computing device 204 isoperating in the second operating mode, the user 208 can cause anautomated assistant to modify a graphical user interface 228 of theportable computing device 204 to include an interactive graphicalelement 230. Specifically, the user 208 can provide a physical gesture220, which can be detected by the portable computing device 204, andinvoke the automated assistant. The interactive graphical element 230that is provided at the graphical user interface 228 can include one ormore selectable elements and/or a natural language output, such as text,pictures, and/or any other graphical output that can be presented at adisplay panel. For instance, in response to detecting the physicalgesture 220, the portable computing device 204 can communicate with theserver device 224 to determine an operating status of one or moredevices that are associated with the user 208. Such operating statusescan include an operating state of a device that is connected to anetwork to which the portable computing device 204 is also connected.For example, the server device 224 can provide contextual data thatindicates a dishwasher is off, and that the client device 206 is playingmusic 212. Based on the contextual data received from the server device224, and/or any other suitable source, the portable computing device 204can generate the interactive graphical element 230. For instance,because the contextual data indicates that the dishwasher is off, theinteractive graphical element 230 can provide a selectable switch withwhich to activate the dishwasher. Additionally, or alternatively,because the contextual data has identified the client device 206 asplaying music, the portable computing device 204 can provide aninteractive graphical element 230 that includes a volume control elementand/or a song selection element, as illustrated in FIG. 2B. In this way,the user 208 does not necessarily need to provide a spoken utterance inorder to adjust the music being played at the client device 206. Thiscan allow the client device 206 to continue streaming musicuninterrupted while simultaneously being controlled by a physicalgesture in which the user 208 does not directly contact and/or directlyspeak to the client device 206.

In some implementations, when the portable computing device 204 isoperating in the second operating mode, the user 208 can use a non-audiofeature(s) of one or more different physical gestures in order tocontrol particular selectable elements displayed at the graphical userinterface 228. For instance, the user can waive their hand to invoke ordismiss the automated assistant. Additionally, or alternatively, theuser 208 can direct their hand or other appendage (e.g. elbow) in acircular motion in order to adjust a position of a portion of aselectable element that has a circular feature, such as the music volumecontrol provided at the interactive graphical element 230. Additionally,or alternatively, the user 208 can perform a lateral motion, or motionthat is parallel to the ground, with an appendage in order to adjust aselectable element that appears to be a switch, such as the on-and-offdishwasher switch provided at the interactive graphical element 230.Additionally, or alternatively, the user can perform a stamping motion,or motion that is perpendicular to the ground, with an appendage inorder to indicate that the user 208 would like to depress a button thatis provided at the interactive graphical element 230, such as the “SKIPSONG” selectable element provided in FIG. 2B.

FIG. 3 illustrates a system 300 for providing a client device with anability to receive non-audio related gestures from a user forcontrolling certain aspects of the client device and/or any other devicethat can be associated with the user. The system 300 can include aclient computing device 302 that is in communication with a servercomputing device 326 and/or a docking apparatus 336. The clientcomputing device 302 can include an automated assistant 318, which canoperate as part of an automated assistant that is provided at one ormore computing devices, such as a first client device (e.g., a cellularphone), a second client device (e.g., a standalone speaker device),and/or a remote computing device 304, such as a server computing device326. A user can interact with the automated assistant 318 via one ormore an assistant interfaces 306, which can include a microphone, acamera, a touch screen display panel, a user interface, a proximitysensor, a touch sensor, a temperature sensor, and/or any other apparatuscapable of being responsive to a user of a computing device. Forinstance, a user can initialize the automated assistant 318 by providinga verbal, textual, and/or a graphical input to the assistant interface306 to cause the automated assistant 318 to perform a function (e.g.,provide data, control a peripheral device, access an agent, submit aquery of a network, etc.). A client computing device 302 that providesat least a portion of the automated assistant 318 can include a displaydevice, which can be a display panel that includes a touch interface forreceiving touch inputs and/or gestures for allowing a user to controlapplications of the client computing device 302 via the touch interface.In some implementations, the client computing device 302 can lack adisplay device, thereby providing an audible user interface output,without providing a graphical user interface output. Furthermore, theclient computing device 302 can provide a user interface, such as amicrophone and/or one or more other sensors, for receiving spokennatural language inputs and/or any other inputs from the user.

The client computing device 302 can be in communication with the servercomputing device 326 over a network, such as the internet. The clientcomputing device 302 can offload computational tasks to the servercomputing device 326, such as speech processing tasks, in order toconserve computational resources at the client computing device 302. Forinstance, in some implementations, the server computing device 326 canhost the automated assistant 318, and the client computing device 302can transmit inputs received at one or more assistant interfaces 306 tothe server computing device 326. However, in some implementations, theautomated assistant 318 can be hosted at the client computing device302. In various implementations, all or less than all aspects of theautomated assistant 318 can be implemented on the client computingdevice 302, and/or at an automated assistant 308 provided at the servercomputing device 326. In some of those implementations, aspects of theautomated assistant 318 are implemented via a local automated assistantof the client computing device 302 and interface with the servercomputing device 326 that implements other aspects of the automatedassistant 318. The server computing device 326 can optionally serve aplurality of users and their associated automated assistants viamultiple threads. In implementations where all or less than all aspectsof the automated assistant 318 are implemented via a local automatedassistant of the client computing device 302, the local automatedassistant can be an application that is separate from an operatingsystem of the client computing device 302 (e.g., installed “on top” ofthe operating system)—or can alternatively be implemented directly bythe operating system of the client computing device 302 (e.g.,considered an application of, but integral with, the operating system).

In some implementations, the automated assistant 308 and/or automatedassistant 318 can include an input processing engine 310, which canemploy multiple different modules for processing inputs and/or outputsfor the client computing device 302. For instance, the input processingengine 310 can include a speech processing module 312 that can processaudio data received at an assistant interface 306 to identify the textembodied in the audio data. The audio data can be transmitted from theclient computing device 302 to the server computing device 326 in orderto preserve computational resources at the client computing device 302.The process for converting the audio data to text can include a speechrecognition algorithm, which can employ neural networks, word2vecalgorithms, and/or statistical models for identifying groups of audiodata corresponding to words or phrases. The text converted from theaudio data can parsed by a data parsing module 314 and made available tothe automated assistant as textual data that can be used to generateand/or identify command phrases from the user. In some implementations,output data provided by the data parsing module 314 can be provided to aparameter module 316 to determine whether the user has provided an inputthat corresponds to a particular action capable of being performed bythe automated assistant and/or an application or agent that is capableof being accessed by the automated assistant. For example, assistantdata 322 can be stored at the server computing device 326 and/or theclient computing device 302 and can include data that defines one ormore actions capable of being performed by the automated assistant 318,as well as parameters necessary to perform the actions. The inputprocessing engine 310 can determine that a user has requested aparticular action be performed, the parameter module 316 can thendetermine one or more parameters for the particular action, and anoutput generating engine 320 can then provide an output to the userbased on the particular action and/or the one or more parameters. Forinstance, in some implementations, in response to a user input, such asa gesture directed at the client computing device 302, the automatedassistant 318 can cause data, which characterizes the gesture, to betransmitted to the server computing device 326 for determining theaction that the user is intending the automated assistant 318 toperform.

In some implementations, the automated assistant 318, the clientcomputing device 302, and/or the server computing device 326 can beresponsive to one or more different types of gestures directed at theclient computing device 302. For instance, when the client computingdevice 302 includes a speaker, a type of gesture that can be used tocontrol the volume can be a two-dimensional gesture (e.g., swiping atouch screen display or otherwise moving an appendage of the user in atleast two-dimensions with or without directly contacting the clientcomputing device 302) or a three-dimensional gesture (e.g., rotating twofingers on the touch screen display or otherwise moving an appendage ofthe user in at least three-dimensions with or without directlycontacting the client computing device 302).

In some implementations, the client computing device 302 can operateaccording to multiple different operating modes as selected by a modeselection engine 328. For instance, when operating in a first operatingmode, the client computing device 302 can invoke an automated assistant318 in response to a spoken trigger phrase or spoken utterance providedby a user. Furthermore, when operating in the first operating mode, theclient computing device 302 can restrict or otherwise limit usage of aproximity sensor of the sensors 324 for invoking the automated assistant318. In other words, despite a user being close enough to the clientcomputing device 302 to cause an output of a proximity sensor to changeor otherwise indicate a presence of the user, the client computingdevice 302 will not invoke the automated assistant 318 despite theoutput of the proximity sensor—at least when the client computing device302 is operating in the first operating mode.

In some implementations, the client computing device 302 can transitioninto the second operating mode, from the first operating mode, based oncontextual data. Contextual data can be client data 330 that isavailable at the client computing device 302, assistant data 322 that isgenerated based on operations of the automated assistant 308 and/orautomated assistant 318, or sensor data 338 that is generated based onoperations of the sensors 334, and/or any combination of theaforementioned data. Client data 330 can include data that characterizesoperations performed by sensors 324, the automated assistant 318, and/orany input or output received or provided at one or more assistantinterfaces 306. For instance, one or more sensors 324 of the clientcomputing device 302 can provide a sensor output that indicates theclient computing device 302 is located a distance away from the user,and/or is located outside of a vicinity of the user that is suitable forthe client computing device 302 to acknowledge a spoken utteranceprovided by the user.

The assistant data 322 can be generated based on one or moreinteractions between the automated assistant 308 and one or more users.For example, the assistant data 322 can characterize an interactionbetween the user and the automated assistant 308, in which the user isrequesting directions while driving. Therefore, the mode selectionengine 428 can use such data to determine that the client computingdevice 302 is currently being, or about to be, transported in a vehicle,and can therefore switch operating modes to a second operating mode sothat the automated assistant 318 can be invoked via user interactionwith a proximity sensor.

The sensor data 338 can be used by the mode selection engine 328 fordetermining whether the client computing device 302 is docked at thedocking apparatus 336. For instance, one or more sensors 334 of thedocking apparatus 336, such as the docking station 340, can detect apresence at the client computing device 302 and cause sensor data 338 tobe generated and transmitted to the client computing device 302. In someimplementations, the docking apparatus 336 can include one or moretransmitters 342 capable of sending and/or receiving data, such as forcommunicating with the client computing device 302. For instance, thedocking apparatus 336 can include a near field communications (NFC)transmitter for broadcasting data, which can be received by the clientcomputing device 302. In response to receiving data from the dockingapparatus 336, the mode selection engine 328 can confirm that the clientcomputing device 302 is docked, and cause the client computing device302 to transition to the second operating mode. As a result, the userwill be able to invoke the automated assistant 318 by moving anappendage near the client computing device 302 simultaneous to theclient computing device 302 being docked at the docking apparatus 336.

In some implementations, certain criteria must be satisfied by thecontextual data in order for the mode selection engine 328 to transitionthe client computing device 302 between operating modes. For example, aclient computing device 302 can include a touch sensor, and criteria fortransitioning the client computing device 302 from the first operatingmode to the second operating mode can be based on whether the user hasinteracted with the touch sensor within a threshold period of time. Ifthe user has not interacted with the touch sensor within a thresholdperiod of time, such as by touching a touch screen display of the clientcomputing device 302, the client computing device 302 can transitioninto a second operating mode in which the automated assistant 318 can beinvoked via interaction between the user and a proximity sensor that isin communication with the client computing device 302. Alternatively, oradditionally, the sensors 334 and/or the sensors 324 can provide sensoroutput that indicates the client computing device 302 is physicallyseparated from the user and/or has been physically separated from theuser for a threshold period of time. In response to such sensor output,the mode selection engine 328 can cause the client computing device 302to transition between operating modes.

In some implementations, a sensor 324 of the client computing device 302can include one or more microphones, capable of being responsive toaudio projected by one or more users when the one or more users arespeaking. A microphone can provide a sensor output that characterizesthe audio being projected by the user, and the mode selection engine 328can cause the client computing device 302 to switch between operatingmodes based on the sensor output of a microphone. Audio data generatedbased on the output of the microphone can provide an indication of anenvironment in which the client computing device 302 is operating, adistance of the client computing device 302 from the user, and/or anyother environmental conditions that may be affecting the clientcomputing device 302.

For instance, based on ambient noise that is exhibited or audible withinan environment of the client computing device 302, the client computingdevice 302 can determine that it is currently located in a vehicle thatis currently driving. Therefore, in response to determining suchenvironmental characteristics, the mode selection engine 328 can causethe client computing device 302 to transition from a first operatingmode to a second operating mode. In this way, the client computingdevice 302 can more readily determine then it is in a vehicle, andtherefore more quickly allow the user to invoke the automated assistant318 by interacting with the proximity sensor of the client computingdevice 302. In some implementations, audio that causes the clientcomputing device 302 to transition between operating modes can beprovided by one or more users, one or more different computing devices,one or more different environmental features, and/or any other objectthat can create audible sound. In some implementations, audio thatcauses the client computing device 302 to transition between operatingmodes can be provided by another computing device that also has accessto an automated assistant. In this way, the client computing device 302is able to transition into a mode where the automated assistant 318 canbe invoked without speaking, thereby eliminating any interruptions tooperations being performed by an automated assistant at anothercomputing device. In response to transitioning to the second operatingmode, the client computing device 302 can provide a graphical userinterface with one or more selectable elements for modifying anoperation of the automated assistant 318 and/or an automated assistantthat is accessible via another computing device. In this way, the usercan perform gestures for interacting with a proximity sensor of theclient computing device 302 in order to control particular operations ofan automated assistant that is operating at a different computingdevice. It should be noted that any of the devices and/or apparatusesdiscussed herein can include one or more transmitters for communicatingdata via Bluetooth, Wi-Fi, LTE, wired, and/or any other protocol forcommunicating data.

FIG. 4 illustrates a method 400 for transitioning a computing devicebetween operating modes based on whether the computing device is dockedat a docking apparatus. The method 400 can be performed by one or moredevices, applications, and/or any other apparatus or module capable ofcontrolling an operating mode of a device. The method 400 can include anoperation 402 of determining whether a device received data from adocking apparatus. A docking apparatus can include an apparatus capableof mechanically supporting at least a portion other device. In someimplementations, a docking apparatus can have additional capabilities ofbroadcasting data so that a device, which has received the broadcasteddata, can determine whether it has been docked with the dockingapparatus. A device, such as a portable computing device (e.g., a cellphone), can include a transmitter that periodically checks forbroadcasted signals (e.g., NFC broadcasts). Therefore, the operation 402can be performed periodically, as indicated by the circular arrowillustrated at FIG. 4.

When data is received by the device and from the docking apparatus, themethod 400 can proceed to an operation 404. The operation 404 caninclude a determination of whether the received data indicates thattheir device is docked. If the data does not indicate that the device isdocked, the method 400 can return to operation 402. If the dataindicates that the device is docked, the method 400 can proceed tooperation 406.

The operation 406 can include causing the device to transition from afirst operating mode to a second operating mode. When operating in thefirst operating mode, the device can limit certain modalities throughwhich an automated assistant can be invoked. When operating in thesecond operating mode, those modalities that were limited in the firstoperating mode can be less limited or otherwise available to the userfor invoking the automated assistant. For example, when operating in thefirst operating mode, the device can restrict a sensor output of aproximity sensor from being used as a basis from which to invoke theautomated assistant. Therefore, when operating in the first operatingmode, the user would not be able to invoke the automated system bycausing a non-audio feature of a physical gesture to be detected by theproximity sensor. However, when operating in the second operating mode,a non-audio feature of a physical gesture can be detected by a proximitysensor that is in communication with the device, and in response todetecting the non-audio feature of the physical gesture, the automatedassistant can be invoked.

The method 400 can proceed to an operation 408 of determining whether agesture was detected by a proximity sensor that is in communication withthe device. A gesture can be one or more different physical actionsperformed by the user. For example, a gesture can be a physical motionof the user that may or may not be intended to create audio, but mayotherwise be detectable by the proximity sensor. As a result of thegesture being performed by the user, the proximity sensor can provide asensor output, which can characterize the performed gesture, and one ormore processors of the device or a remote device can process the sensoroutput to determine whether the gesture corresponds to a request toperform a particular action that can be performed by an automatedassistant.

When the gesture has not been detected by the proximity sensor, themethod 400 can proceed to an operation 412 of determining whether thedevice is still docked at the docking apparatus. If the device is stilldocked, the method 400 can return to the operation 408. If the device isno longer docked, the method 400 can proceed to an operation 414. Theoperation 414 can include causing the device to operate according to thefirst operating mode. The device can be transitioned to the firstoperating mode when the device is no longer docked, in order that theproximity sensor can be employed for other functions of the device,rather than invoking the automated assistant. Adapting the usage ofparticular modalities through which the automated assistant can beinvoked allows for a more effective and efficient usage of suchmodalities in particular environments. For instance, when the device isno longer docked, the proximity sensor can be employed to determine howproximate the device is to the user. For example, should the user placetheir device in their pocket, the device can operate in the firstoperating mode so that the proximity sensor does not trigger theautomated assistant, but rather can trigger a vibration mode so that theuser receives application notifications through vibration. Furthermore,limiting how the automated assistant is invoked can eliminate accidentaltriggering of the automated assistant, which can reduce unnecessarypower consumption spikes and preserve network bandwidth.

When, at operation 408, a gesture is detected by the proximity sensor ofthe device, the method 400 can proceed to an operation 410. Theoperation 410 can include causing an automated assistant to perform anaction based on the gesture. For example, a user can trigger theautomated assistant when the device is docked within their vehicle byhovering their hand at, or within, a threshold distance from the device.In this way, should the vehicle have other occupants or other devicescreating audio, the user would not necessarily have to rely onprojecting a spoken utterance to invoke the automated assistant. Rather,the user can choose to invoke the automated assistant by performing agesture that is detected by the proximity sensor or providing a spokenutterance that is detected by a microphone of the computing device, thedocking apparatus, and/or a vehicle computing device that is integral tothe vehicle. In some implementations, one or more different gestures cancause the automated assistant to perform one or more different actionswhen the device is operating in the second operating mode. Furthermore,when the device is operating in the first operating mode, the automatedassistant can perform the one or more actions in response to the userperforming one or more other gestures. In other words, the automatedassistant can be responsive to a unique set of gestures for eachoperating mode. Additionally, or alternatively, one or more sensors candetect gestures for invoking or controlling the automated assistant whenoperating in the first operating mode, and one or more other sensors canbe used to detect gestures for invoking or controlling the automatedassistant when operating in the second operating mode.

FIG. 5 illustrates in method 500 for causing an automated assistant toperform a particular action when a computing device is operating in amode via which the automated assistant is capable of being responsive tonon-audio features of a physical gesture. The method 500 can beperformed by one or more applications, devices, and/or any otherapparatus or module capable of interacting with an automated assistant.The method 500 can include an operation 502 of causing a portablecomputing device to operate according to a first operating mode. Whenoperating in the first operating mode, the automated assistant can beconfigured to be invoked according to a spoken utterance from a user.For example, the spoken utterance can be a trigger phrase such as,“Assistant,” and/or any other spoken utterance that is based on anatural language. Additionally, or optionally, when operating in thefirst operating mode, the automated assistant can be unresponsive to atleast an initial invocation attempt that is exclusively non-audio, suchas movement of an appendage or other motion created by the user.

The method 500 can further include an operation 504 of determining thatthe portable computing device is situated such that a sensor incommunication with the portable computing device is capable of detectinga gesture performed by the user. The determination at operation 504 canbe determined based on contextual data that is accessible to theportable computing device. The contextual data can be provided from oneor more different sources, such as one or more different sensors,computing devices, and/or any other device capable of providing data. Insome implementations, the contextual data can characterize an operatingstatus of one or more devices that are on a common network with theportable computing device. Alternatively, or additionally, thedetermination at operation 504 can be based on operational data that isassociated with the portable computing device. The operational data cancharacterize an operation of one or more devices and/or applicationsthat are provided at the portable computing device, such as a controlapplication and/or one or more sensors. The one or more sensors of theportable computing device can indicate an orientation of the portablecomputing device, a velocity, a trajectory, acceleration, an amount oflight within an environment of the portable computing device, atemperature, and/or any other information that can be characterized by acomputing device. For instance, a sensor output can be used by theportable computing device to generate trajectory data that characterizesa velocity and/or a direction in which the portable computing device ismoving.

The operational data and/or the contextual data can indicate that theportable computing device is physically separated from the user and/orat a location that is within a distance in which a sensor of theportable computing device can be responsive to a physical gestureperformed by the user. Alternatively, or additionally, the contextualdata and/or the operational data can indicate that the portablecomputing device is located within an environment that includes audio,such as ambient noise, that would interrupt a spoken utterance providedby the user to the portable computing device. For instance, an output ofa microphone can detect noise created by one or more other personsand/or one or more other computing devices, and determine, based on alevel of noise, that the noise would interfere with the portablecomputing device detecting a spoken utterance from the user.

The method 500 can further include an operation 506 of causing theportable computing device to operate according to a second operatingmode in which the automated assistant can be invoked via the gesturebeing detected by the sensor. As a result of operating in the secondoperating mode, interference that would otherwise occur and interruptthe portable computing device, when detecting a spoken utterance, can beobviated by causing the portable computing device to be responsive tonon-audio features of particular gestures. For example, an amount ofspace occupied by an appendage of the user, a change in an amount ofspace occupied by an appendage of the user, a velocity and/or anacceleration of an appendage of the user, a shape and/or a change inshape of one or more appendages of the user, and/or any other physicalattribute of the user can be detected by the portable computing devicewhen operating in the second operating mode.

The method 500 can also include an operation 508 of determining that theproximity sensor has detected the user performing the gesture. Thegesture can be, for example, a hand waving motion performed by the user.In some implementations, the gesture can be a movement of the mouth ofthe user, wherein the portable computing device is responsive to thecharacteristics of the movement of the mouth of the user and not anyaudio created in response to the user moving their mouth. In this way,the user can audibly mouth a physical gesture that is detected by theportable computing device when the portable computing device isoperating in the second operating mode. In some implementations, theportable computing device can include one or more proximity sensors,and/or an array of proximity sensors, thereby allowing the automatedassistant to be responsive to changes in position of an appendage of theuser when the user has placed their appendage within a distance from theportable computing device that allows the proximity sensors to beresponsive to the movement of the appendage.

The method 500 can also include an operation 510 of causing theautomated assistant to be invoked at the portable computing deviceand/or a separate computing device. For example, the user may be withinan environment such as their home, which can include multiple computingdevices that have access to the automated assistant. If one particulardevice is playing music, another computing device can detect the musicand transition into the second operating mode. A physical gesture can bedetected by the other computing device in order to make adjustments tothe music playing operation at the particular computing device. In thisway, the user does not necessarily have to project an audible spokenutterance in order to manipulate the music playing operation, but,rather, can rely on at least one other device transitioning to thesecond operating mode, thereby allowing the user to perform gesturesthat have non-audio features for controlling the music playbackoperation.

Optionally, the method 500 can include an operation 512 of causing theautomated assistant to provide a natural language output via anautomated assistant interface of the portable computing device and/orthe separate computing device. For example, when the portable computingdevice is operating in the second operating mode, and the user performsa physical gesture to control the automated assistant, the physicalgesture can cause the automated assistant to provide a natural languageoutput at the portable computing device and/or the separate computingdevice. The natural language output can be, for example, an audio output(e.g., a portion of an audible dialog or other natural language speech)from a speaker of the portable computing device or the separatecomputing device, a graphical display element provided at is displaypanel of the portable computing device or the separate computing device,and/or any other type of natural language output that can be providedvia a modality of a computing device. In this way, although an automatedassistant may be providing an output at the separate computing device,the user can perform a physical gesture directed at the portablecomputing device in order to control one or more operations or actionsbeing performed by the automated assistant at the separate computingdevice. This can allow for the distribution of processing of gesturesacross multiple devices, rather than relying on a single device that isalready actively performing an operation to process or otherwise beresponsive to such gestures. This can preserve computational resources,as well as reduce power spikes that can occur at a particular devicethat has been designated to exclusively be responsive to, or process,gestures, rather than allowing multiple other computing devices to beresponsive to, or process, various types of gestures.

FIG. 6 is a block diagram of an example computer system 610. Computersystem 610 typically includes at least one processor 614 whichcommunicates with a number of peripheral devices via bus subsystem 612.These peripheral devices may include a storage subsystem 624, including,for example, a memory 625 and a file storage subsystem 626, userinterface output devices 620, user interface input devices 622, and anetwork interface subsystem 616. The input and output devices allow userinteraction with computer system 610. Network interface subsystem 616provides an interface to outside networks and is coupled tocorresponding interface devices in other computer systems.

User interface input devices 622 may include a keyboard, pointingdevices such as a mouse, trackball, touchpad, or graphics tablet, ascanner, a touchscreen incorporated into the display, audio inputdevices such as voice recognition systems, microphones, and/or othertypes of input devices. In general, use of the term “input device” isintended to include all possible types of devices and ways to inputinformation into computer system 610 or onto a communication network.

User interface output devices 620 may include a display subsystem, aprinter, a fax machine, or non-visual displays such as audio outputdevices. The display subsystem may include a cathode ray tube (CRT), aflat-panel device such as a liquid crystal display (LCD), a projectiondevice, or some other mechanism for creating a visible image. Thedisplay subsystem may also provide non-visual display such as via audiooutput devices. In general, use of the term “output device” is intendedto include all possible types of devices and ways to output informationfrom computer system 610 to the user or to another machine or computersystem.

Storage subsystem 624 stores programming and data constructs thatprovide the functionality of some or all of the modules describedherein. For example, the storage subsystem 624 may include the logic toperform selected aspects of method 400, method 500, and/or to implementone or more of computing device 110, automated assistant 112, modeselection engine 114, server device 224, client device 206, servercomputing device 326, client computing device 302, docking apparatus336, and/or any other device, application, and/or operation discussedherein.

These software modules are generally executed by processor 614 alone orin combination with other processors. Memory 625 used in the storagesubsystem 624 can include a number of memories including a main randomaccess memory (RAM) 630 for storage of instructions and data duringprogram execution and a read only memory (ROM) 632 in which fixedinstructions are stored. A file storage subsystem 626 can providepersistent storage for program and data files, and may include a harddisk drive, a floppy disk drive along with associated removable media, aCD-ROM drive, an optical drive, or removable media cartridges. Themodules implementing the functionality of certain implementations may bestored by file storage subsystem 626 in the storage subsystem 624, or inother machines accessible by the processor(s) 614.

Bus subsystem 612 provides a mechanism for letting the variouscomponents and subsystems of computer system 610 communicate with eachother as intended. Although bus subsystem 612 is shown schematically asa single bus, alternative implementations of the bus subsystem may usemultiple busses.

Computer system 610 can be of varying types including a workstation,server, computing cluster, blade server, server farm, or any other dataprocessing system or computing device. Due to the ever-changing natureof computers and networks, the description of computer system 610depicted in FIG. 6 is intended only as a specific example for purposesof illustrating some implementations. Many other configurations ofcomputer system 610 are possible having more or fewer components thanthe computer system depicted in FIG. 6.

In situations in which the systems described herein collect personalinformation about users (or as often referred to herein,“participants”), or may make use of personal information, the users maybe provided with an opportunity to control whether programs or featurescollect user information (e.g., information about a user's socialnetwork, social actions or activities, profession, a user's preferences,or a user's current geographic location), or to control whether and/orhow to receive content from the content server that may be more relevantto the user. Also, certain data may be treated in one or more waysbefore it is stored or used, so that personal identifiable informationis removed. For example, a user's identity may be treated so that nopersonal identifiable information can be determined for the user, or auser's geographic location may be generalized where geographic locationinformation is obtained (such as to a city, ZIP code, or state level),so that a particular geographic location of a user cannot be determined.Thus, the user may have control over how information is collected aboutthe user and/or used.

While several implementations have been described and illustratedherein, a variety of other means and/or structures for performing thefunction and/or obtaining the results and/or one or more of theadvantages described herein may be utilized, and each of such variationsand/or modifications is deemed to be within the scope of theimplementations described herein. More generally, all parameters,dimensions, materials, and configurations described herein are meant tobe exemplary and that the actual parameters, dimensions, materials,and/or configurations will depend upon the specific application orapplications for which the teachings is/are used. Those skilled in theart will recognize, or be able to ascertain using no more than routineexperimentation, many equivalents to the specific implementationsdescribed herein. It is, therefore, to be understood that the foregoingimplementations are presented by way of example only and that, withinthe scope of the appended claims and equivalents thereto,implementations may be practiced otherwise than as specificallydescribed and claimed. Implementations of the present disclosure aredirected to each individual feature, system, article, material, kit,and/or method described herein. In addition, any combination of two ormore such features, systems, articles, materials, kits, and/or methods,if such features, systems, articles, materials, kits, and/or methods arenot mutually inconsistent, is included within the scope of the presentdisclosure.

I claim:
 1. A method implemented by one or more processors, the methodcomprising: causing a portable computing device to operate in a firstoperating mode, wherein, when the portable computing device is operatingin the first operating mode, the portable computing device is configuredto: invoke an automated assistant in response to detection of a spokentrigger phrase, and restrict usage of a proximity sensor, which is incommunication with the portable computing device, for invoking theautomated assistant; processing contextual data that characterizes acurrent context of the portable computing device, wherein the contextualdata is generated based on sensor output from one or more sensors of theportable computing device, wherein processing the contextual dataincludes determining, based on processing the contextual data, that anenvironment of the portable computing device is receiving audio, whereinthe audio is being projected by another computing device; and causingthe portable computing device to operate in a second operating mode inresponse to the processing of the contextual data satisfying one or morecriteria, wherein the one or more criteria include determining that theenvironment of the portable computing device is receiving audio datathat is being projected by the other computing device, and wherein, whenthe portable computing device is operating in the second operating mode,the portable computing device is configured to invoke the automatedassistant in response to the proximity sensor indicating that a physicalgesture was performed by a user; and when the portable computing deviceis operating in the second operating mode and the proximity sensor ofthe portable computing device indicates that the physical gesture wasperformed by the user: causing the automated assistant to be invoked inresponse to the proximity sensor indicating that the physical gesturewas performed by the user.
 2. The method of claim 1, wherein the one ormore sensors include a transmitter that is connected to a dockingapparatus and wherein processing the contextual data further includes:determining that the portable computing device is within a minimaldistance for receiving broadcasted data from the transmitter.
 3. Themethod of claim 1, wherein the portable computing device includes atouch sensor and wherein processing the contextual data includes:determining that the touch sensor has not received direct physicalcontact from the user for a threshold period of time.
 4. The method ofclaim 1, wherein processing the contextual data includes determining,based on the sensor output, that the portable computing device has beenstationary at a location that is physically separated from the user fora threshold period of time.
 5. The method of claim 1, wherein processingthe contextual data includes determining, based on processing thecontextual data, that an environment of the portable computing device isreceiving audio being projected by another person, and wherein causingthe portable computing device to operate in the second operating mode isfurther in response to determining that the environment of the portablecomputing device is receiving audio that is being projected by the otherperson.
 6. The method of claim 1, further comprising: causing, inresponse to the proximity sensor indicating that the physical gesturewas performed by the user, a touch screen display of the portablecomputing device to provide an interactive graphical element thatincludes natural language text, wherein the interactive graphicalelement is configured to modify an operation of the other computingdevice in response to the user providing a different gesture forselecting the interactive graphical element.
 7. The method of claim 6,wherein the other computing device is performing the operation when theportable computing device is operating in the second operating mode. 8.A method implemented by one or more processors, the method comprising:causing a portable computing device to operate according to a firstoperating mode in which the portable computing device is configured toinvoke an automated assistant in response to a user providing a spokenutterance and restrict usage of a proximity sensor for invoking theautomated assistant; receiving, while the portable computing device isoperating in the first operating mode, data indicating that the portablecomputing device is physically situated such that the proximity sensor,which is integral to the portable computing device, is capable ofdetecting a physical gesture performed by the user, wherein the dataindicating that the portable computing device is physically situatedsuch that the proximity sensor is capable of detecting the physicalgesture performed by the user includes location data that characterizesa location of the portable computing device relative to the user or idletime data that characterizes an amount of time that the user has ceasedproviding an input to the portable computing device; causing, inresponse to receiving the data, the portable computing device to operateaccording to a second operating mode in which the automated assistant isconfigured to be invoked in response to the proximity sensor detecting anon-audio feature of the physical gesture performed by the user; whenthe proximity sensor detects the non-audio feature of the physicalgesture performed by the user: causing the automated assistant toprovide a natural language output via an automated assistant interfaceof the portable computing device.
 9. The method of claim 8, furthercomprising: monitoring by the portable computing device a sensor outputof the proximity sensor more frequently when operating in the secondoperating mode than the first operating mode.
 10. The method of claim 8,wherein the natural language output is at least a portion of audibledialog and the automated assistant interface is a speaker that isconnected to the portable computing device.
 11. The method of claim 8,wherein the natural language output is text that is provided at aselectable element, which is displayed at a touch screen display panelof the portable computing device.
 12. The method of claim 8, wherein thedata indicating that the portable computing device is physicallysituated such that the proximity sensor is capable of detecting thephysical gesture performed by the user includes trajectory data thatcharacterizes a velocity or a direction in which the portable computingdevice is moving.
 13. The method of claim 8, further comprising:monitoring a sensor output of the proximity sensor to determine whetherthe non-audio feature of the physical gesture has been performed by theuser, wherein the non-audio feature of the physical gesture includespositioning an appendage of the user proximate enough to the portablecomputing device to cause a change in the sensor output of the proximitysensor.